Image recognition

ABSTRACT

In a data processing system for recognising a subject image ( 1 ), first processing means ( 2 ) is arranged to derive from the subject image ( 1 ) at least one graph having a plurality of points representing topographical data of the subject image ( 1 ). Second processing means ( 3 ) is arranged to create, for each of those points, attribute data representing at least one attribute of the subject image ( 1 ) corresponding to the respective point, which attribute is in addition to the topographical data. At least one correlation matrix memory ( 7 ) is arranged to provide at least part of storage means ( 4 ) arranged to store data of stored images; comparison means ( 5 ) arranged to compare the data of the subject image ( 1 ) with the stored data of the stored images; and identifying means ( 6 ) arranged to identify matches between data of the subject image ( 1 ) and the stored data. The data processing system may be used with advantage in recognising natural images e.g. human faces.

This invention relates to the recognition of images, and is concerned,particularly although exclusively, with the recognition of naturalimages.

By “natural image” is meant an image of an object that occursnaturally—for example, an optical image such as a photograph, as well asimages of other wavelengths—such as x-ray and infra-red, by way ofexample. The natural image may be recorded and/or subsequently processedby digital means, but is in contrast to an image—or image data—that isgenerated or synthesised by computer or other artificial means.

The recognition of natural images can be desirable for many reasons. Forexample, distinctive landscapes and buildings can be recognised, toassist in the identification of geographical locations. The recognitionof human faces can be useful for identification and security purposes.The recognition of valuable animals such as racehorses may be veryuseful for identification purposes.

Various attempts have been made to provide systems and methods torecognise faces, for example. Generally, this involves comparing acurrent optical image with a number of stored optical images, andlooking for a match. Although theoretically possible, this gives rise toa number of practical problems. For example, the current optical imagemay have numerous small differences from the stored optical image withwhich it should match—this is a factor that does not usually arise withsynthetic images. Large amounts of computer processing however arerequired and, even with relatively powerful modern computing systems,the matching process tends to take an unacceptably long time and produceunacceptably low recognition success.

Certain preferred embodiments of the present invention aim to providesystems and methods for matching a natural image with a respective oneof a large number of stored images.

Other preferred embodiments of the present invention aim to providesystems and methods for matching both natural and synthetic images witha respective one of a large number of stored images.

According to one aspect of the present invention, there is provided adata processing system for recognising a subject image, the systemcomprising:

-   -   a. first processing means arranged to derive from the subject        image at least one graph having a plurality of points        representing topographical data of the subject image;    -   b. second processing means arranged to create, for each of said        points, attribute data representing at least one attribute of        the subject image corresponding to the respective point, which        attribute is in addition to said topographical data; and    -   c. at least one correlation matrix memory that is arranged to        provide at least part of:    -   d. storage means arranged to store data of stored images;    -   e. comparison means arranged to compare the data of the subject        image with the stored data of the stored images; and    -   f. identifying means arranged to identify matches between data        of the subject image and said stored data.

Preferably, said identifying means is arranged to calculate, for each ofa plurality of points of the subject image, and for the or each saiditem of attribute data pertaining to that point, potential matches fromsaid stored data for said item of attribute data, and support from othersuch points for each of said potential matches, and then toprogressively remove potential matches of least support.

Preferably, said attribute data comprises data representing at least oneof colour, texture and curvature at the respective said point.

Preferably, for each said point, said attribute data is relational data,representing a difference in corresponding values of a common attributeas between the respective point and another one of said points.

Preferably, said subject image is a natural image.

Preferably, said natural image is an image of a human face.

In another aspect, the invention provides a method of recognising asubject image, comprising the steps of:

-   -   a. deriving from the subject image at least one graph having a        plurality of points representing topographical data of the        subject image;    -   b. creating for each of said points attribute data representing        at least one attribute of the subject image corresponding to the        respective point, which attribute is in addition to said        topographical data; and    -   c. at least partly by means of a correlation matrix memory:    -   d. storing data of stored images;    -   e. comparing the data of the subject image with the stored data        of the stored images; and    -   f. identifying matches between data of the subject image and        said stored data.

Such a method may be carried out by a system according to any of thepreceding aspects of the invention.

In another aspect, the invention provides a data processing system forrecognising a natural image, the system comprising:

-   -   a. processing means arranged to derive from the natural image at        least one graph having a plurality of points representing data        of the natural image; and    -   b. at least one correlation matrix memory that is arranged to        provide at least part of:    -   c. storage means arranged to store data of stored images;    -   d. comparison means arranged to compare the data of the natural        image with the stored data of the stored images; and    -   e. identifying means arranged to identify matches between data        of the natural image and said stored data.

Preferably, said identifying means is arranged to calculate, for each ofa plurality of points of the natural image, potential matches from saidstored data for an item of data at that point, and support from othersuch points for each of said potential matches, and then toprogressively remove potential matches of least support.

Preferably, for each said point, said data is relational data,representing a difference in corresponding values of a common propertyas between the respective point and another one of said points.

Preferably, said data is positional data.

In another aspect, the invention provides a method of recognising anatural image, comprising the steps of:

-   -   a. deriving from the natural image at least one graph having a        plurality of points representing data of the natural image; and    -   b. at least partly by means of a correlation matrix memory:    -   c. storing data of stored images;    -   d. comparing the data of the natural image with the stored data        of the stored images; and    -   e. identifying matches between data of the natural image and        said stored data.

Such a method may be carried out by a system according to any of thepreceding aspects of the invention.

For a better understanding of the invention, and to show how embodimentsof the same may be carried into effect, reference will now be made, byway of example, to the accompanying diagrammatic drawings, in which:

FIG. 1 illustrates one example of a system for recognising naturalimages of human faces, in accordance with one embodiment of theinvention;

FIG. 2 illustrates a number of graphical points and relational databetween those points;

FIG. 3 illustrates computation of model support between points;

FIG. 4 illustrates calculation of model support at a point; and

FIG. 5 illustrates implementation by way of a correlation matrix memory.

In FIG. 1, a subject image that is a natural image, that is to say, inthis example, an optical image of a human head, is reduced to digitaldata and input to a first processing means 2 which derives from theimage data at least one graph having a plurality of points representingtopographical data of the original image. Methods of doing this are wellknown to those skilled in the art—for example, by use of stereoalgorithms, structured light, and so on. In many examples, a pluralityof graphs may be derived, but for ease of explanation, it will beassumed in the following that there is just one graph. The graph has aplurality of points representing relationships between two variables—forexample, x and y coordinates—and values of points on the graph may beconveniently stored as a table.

The image data is also input to a second processing means 3 whichcreates, for each of the points of the graph, attribute datarepresenting at least one attribute of the natural image correspondingto the respective point, which attribute is in addition to thetopographical data. For example, such attributes may be one or more ofcolour, texture and curvature.

A storage means 4 stores topographical and attribute data for aplurality of known, stored images.

A comparison means 5 compares the data of the original image 1 with thestored data of the stored images.

An identifying means 6 identifies matches between data of the originalimage and that of the stored images.

The storage means 4, comparison means 5 and identifying means 6 are allat least partly provided by a correlation matrix memory (CMM) 7.

For a description of a CMM and how it may be used to match incompletedata with stored data, the reader is referred to our prior patentpublication WO 01/01345. Many of the techniques that are employed inthat publication may be utilised in embodiments of the presentinvention.

In order to compare the data of the original image with that of thestored images, a technique called Relaxation By Elimination (RBE) isutilised, and this will be described with reference to FIGS. 2 to 5.

We assume for this explanation that the natural image to be recognisedhas i data points Ni, and that the stored images form a set of j modelsMj with which data at points Ni is to be compared. For simplicity, inthis example, the models Mj each have the same number i of data points,and for ease of reference, the data points of the models Mj will becalled “model points”. However, more generally, the models Mj may havediffering numbers of model points.

FIG. 2 illustrates a simple example where i=3, giving three data pointsN1 to N3 of the image to be identified, and their inter-point distances,D_(1,3) etc.

For each data point Ni, there is created a list Mj of models having datafor corresponding properties or attributes at model points, whichmatches the data at data point Ni. Such property or attribute could beposition, inter-point distance, colour, texture, curvature, etc.Distances from each node Ni to, say, a centre of mass, couldalternatively or additionally be utilised.

Thus, each data point Ni is “seeded” with a respective list of models Mjthat could conceivably fit the initial item of data at the data pointNi. Models having data at model points that could not possibly match thecorresponding item of data at the corresponding data point Ni arediscarded—which greatly assists processing speed.

The system then applies knowledge of inter-point distances. It visitseach data point Ni, and for each checks the knowledge held at other datapoints to find any support for the models at Ni. That is, if a model Mjsupports the inter-point distance D_(1,3) at data point N1, then we ask:is the same model listed at data point N3 to support the same distanceD_(1,3)? If so, that model at data point N1 is supported at data pointN3.

The computation of model support to node N3 by nodes N1 and N2 isvisualised in FIG. 3, and expressed below. For each data point, i, in N.  For all j, visit each data point, j, in N (not including i)    Calculate the distance, D_(i,j,) between the two points, i and j.    Find which of the current models, M′_(j), support     points at thedistance     D_(i,j).   endfor endfor

This results in a support vector, M_(ji) that shows how the data pointsupports the data point i, given the models at j and the distance Di,j.This is where use of the CMM is particularly advantageous. The systemthen looks at each data point Ni and computes the support for its modelsthat is given from other nodes.

The computation of model support to node N3 by other nodes is visualisedin FIG. 4, and expressed below. For each point, i, in N.   Sum thesupport for all model points, M_(i) to get   the raw support M_(i)^(raw) at the node i.   Threshold M_(i) ^(raw) at a level T to   get abinary model support vector, Mi. endfor

The system then eliminates all models that have little support. In thiscase, it is performed over all data points Ni and all models Mj at eachdata point. In effect, this is by setting T to an appropriate value.Over all points p in NxM   Find the model with the least support, anddelete it. endfor

The above processes are reiterated until all models have a support thatexceeds the threshold T. This may leave one or a plurality of modelswith the same support. False matches can then be eliminated by usingconventional geometric matching (or other techniques) on what is, now, asmall set of candidate models.

The process halts when the support at all nodes fails to change. Inpractice, this may not be the lowest energy state of the system, in thata large number of nodes may remain With high support. In this case a‘kick start’ can be given to the node with the highest entropy, byincreasing T at that node, effectively removing the least supportedmodel at that node.

In practice, all neighbours of a given node are selected to provideinformation to update that node. It is possible to use a subset of theneighbours, but this leads to slower convergence.

The approach uses a process of removing points that do not get supportfrom other nodes. The motivation for this is based on the observationthat it is simpler and more reliable to eliminate all models that haveno support, and to let this knowledge propagate, than to select modelsthat have the highest support as found in other relaxation basedmethods.

The CMM 7 is used to store information concerning “which points supportwhich models”.

At each data point Ni there is a list of models Mj that have modelpoints that match the current data point. A search must be made of themodel base to find models that are supported. The input of the CMM is a2D matrix shown in FIG. 5, which codes currently supported models, Mj,against, say, the inter-point distance of interest, Dj,i. This is inputto the CMM, which then looks up to find the models that match andoutputs a raw vector O_(i) ^(raw) that expresses this. This vector isthen thresholded at a level Y to obtain a binary vector giving themodels supported at data point j from data point i, given as M^(j) _(i).This information is sent to the data point currently being evaluatedwhere it is combined as given above. The threshold level, Y, isdetermined from the number of bits set in the input to the CMM, which iscontrolled by the number of currently matching models. In practice Y canbe reduced. The parameterisation of the memory is derived from analysisof CMM storage ability.

The above description is based, for the sake of simplicity, on a singledata value at each data point Ni—for example, inter-point distance Di,j.However, to provide more effective matching or recognition for naturalimages, further attribute data is also considered. For example, at eachdata point Ni, such attribute data values as colour, texture andcurvature may be considered. Preferably, such attribute data values areexpressed in relational terms—that is, for example, “data point N3 isredder than data point N1”—or “has smoother textured than”—or “has lowercurvature than”. These relational attribute data values can convenientlybe superposed upon or otherwise combined with the inter-point distancevalues Di,j, to provide very distinctive links between data points Ni,and thereby speed matching and recognition.

Thus, for example, in FIG. 5, the 1-dimensional array Dn may be replacedby a multi-dimensional matrix, containing a plurality of relationalattribute data values, in addition to the inter-point data.

If there are two (or more) one-dimensional arrays such as Dn forrespective attributes D1 and D2, a matrix can be created from theirouter product, to replace the illustrated single array Dn—or one canadopt simple superposition of data (logical OR-ing of the two arrays).

It will be appreciated that the CMM provides a particularly convenientway to superpose data values and, in this respect, the reader isreferred to our prior patent publication WO 01/01345, where variousaspects and methods for the superposition of data items for both memorytraining and memory recall are disclosed. The above-mentioned patentpublication also describes how various possible matches to query datacan be reduced, and subsequent matching done by other techniques, on asmall number of candidate matches.

Although in the illustrated example, the data points Ni are labelled, itmay be possible to dispense with such labelling, thereby reducing theamount of data to be processed and correspondingly increasing the speedof processing, without, rather surprisingly, losing a great deal inaccuracy.

Whilst, in the above example, a single CMM is shown and described, inpractice an array of CMM's may be utilised—as disclosed in ourabove-mentioned patent publication, for example, which also disclosesvarious techniques for compressing the size of CMMs.

Preferred embodiments of the invention may be utilised for recognisingnatural images—for example, human faces—from a large collection ofstored images, in a reasonably speedy manner.

Other embodiments of the invention may be adapted to recognisesynthesised images in a similar manner.

For the recognition of natural images, positional data only, or anothersingle property or attribute, may be utilised for matching the naturalimage with stored images.

In this specification, the verb “comprise” has its normal dictionarymeaning, to denote non-exclusive inclusion. That is, use of the word“comprise” (or any of its derivatives) to include one feature or more,does not exclude the possibility of also including further features.

The reader's attention is directed to all and any priority documentsidentified in connection with this application and to all and any papersand documents which are filed concurrently with or previous to thisspecification in connection with this application and which are open topublic inspection with this specification, and the contents of all suchpapers and documents are incorporated herein by reference.

All of the features disclosed in this specification (including anyaccompanying claims, abstract and drawings), and/or all of the steps ofany method or process so disclosed, may be combined in any combination,except combinations where at least some of such features and/or stepsare mutually exclusive.

Each feature disclosed in this specification (including any accompanyingclaims, abstract and drawings), may be replaced by alternative featuresserving the same, equivalent or similar purpose, unless expressly statedotherwise. Thus, unless expressly stated otherwise, each featuredisclosed is one example only of a generic series of equivalent orsimilar features.

The invention is not restricted to the details of the foregoingembodiment(s). The invention extends to any novel one, or any novelcombination, of the features disclosed in this specification (includingany accompanying claims, abstract and drawings), or to any novel one, orany novel combination, of the steps of any method or process sodisclosed.

1. A data processing system for recognising a subject image, the systemcomprising: a. a first processing means arranged to derive from thesubject image at least on graph having a plurality of pointsrepresenting topographical data of the subject image; b. secondprocessing means arranged to create, for each of said points, attributedata representing at least one attribute of the subject imagecorresponding to the respective point, which attribute is in addition tosaid topographical data; and c. at least one correlation matrix memorythat is arranged to provide at least part of: d. storage means arrangedto store data of stored images; e. comparison means arranged to comparethe data of the subject image with the stored data of the stored images;and f. identifying means arranged to identify matches between data ofthe subject image and said stored data.
 2. A system according to claim1, wherein said identifying means is arranged to calculate, for each ofa plurality of points of the subject image, and for the or each saiditem of attribute data pertaining to that point, potential matches fromsaid stored data for said item of attribute data, and support from othersuch points for each of said potential matches, and then toprogressively remove potential matches of least support.
 3. A systemaccording to claim 1, wherein said attribute data comprises datarepresenting at least one of colour, texture and curvature at therespective said point.
 4. A system according to claim 1, wherein, foreach said point, said attribute data is relational data, representing adifference in corresponding values of a common attribute as between therespective point and another one of said points.
 5. A system accordingto claim 1, wherein said subject image is a natural image.
 6. A systemaccording to claim 5, wherein said natural image is an image of a humanface.
 7. A method of recognising a subject image, comprising the stepsof: a. deriving from the subject image at least one graph having aplurality of points representing topographical data of the subjectimage; b. creating for each of said points attribute data representingat least one attribute of the subject image corresponding to therespective point, which attribute is in addition to said topographicaldata; and c. at least partly by means of a correlation matrix of memory:d. storing data of stored images; e. comparing the data of the subjectimage with the stored data of the stored images; and f. identifyingmatches between data of the subject image and said stored data.
 8. Amethod according to claim 7, when carried out by a system according to adata processing system for recognising a subject image, the systemcomprising: a. a first processing means arranged to derive from thesubject image at least on graph having a plurality of pointsrepresenting topographical data of the subject image; b. secondprocessing means arranged to create, for each of said points, attributedata representing at least one attribute of the subject imagecorresponding to the respective point, which attribute is in addition tosaid topographical data; and c. at least one correlation matrix memorythat is arranged to provide at least part of: d. storage means arrangedto store data of stored images; e. comparison means arranged to comparethe data of the subject image with the stored data of the stored images;and f. identifying means arranged to identify matches between data ofthe subject image and said stored data.
 9. A data processing system forrecognising a natural image, the system comprising: a. processing meansarranged to derive from the natural image at least one graph having aplurality of points representing data of the natural image; and b. atleast one correlation matrix that is arranged to provide at least partof: c. storage means arranged to store data of stored images; d.comparison means arranged to compare the data of the natural image withthe stored data of the stored images; and e. identifying means arrangedto identify matches between data of the natural image and said storeddata.
 10. A system according to claim 9, wherein said identifying meansis arranged to calculate, for each of a plurality of points of thenatural image, potential matches from said stored data for an item ofdata at that point, and support from each other such points for each ofsaid potential matches, and then to progressively remove potentialmatches of least support.
 11. A system according to claim 9, wherein,for each said point, said data is relational data, representing adifference in corresponding values of a common property as between therespective point and another one of said points.
 12. A system accordingto claim 9, wherein said data is positional data.
 13. A method ofrecognising a natural image, comprising the steps of: a. deriving fromthe natural image at least one graph having a plurality of pointsrepresenting data of the natural image; and b. at least partly by meansof a correlation matrix memory: c. storing data of stored images; d.comparing the data of the natural image with the stored data of thestored images; and e. identifying matches between data of the naturalimage and said stored data.
 14. A method according to claim 13, whencarried out by a data processing system for recognising a natural imagethe system comprising the steps of: a. processing means arranged toderive from the natural image at least one graph having a plurality ofpoints representing data of the natural image; and b. at least onecorrelation matrix that is arranged to provide at least part of: c.storage means arranged to store data of stored images; d. comparisonmeans arranged to compare the data of the natural image with the storeddata of the stored images; and e. identifying means arranged to identifymatches between data of the natural image and said stored data. 15.-16.(canceled)